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Abstract 

Growth  in  government  investment,  academic  research,  and  commercial  question 
answering  (QA)  systems  is  motivating  a  need  for  increased  planning  and  coordination. 
The  internationalization  of  QA  research,  and  the  need  to  move  toward  a  common 
understanding  of  resources,  tasks  and  evaluation  methods  provided  motivate  a  need  to 
facilitate  more  rapid  and  efficient  progress.  This  paper  characterizes  a  range  of  question 
answering  systems  and  provides  an  initial  roadmap  for  future  research,  including  a  list  of 
existing  resources  and  ones  under  development.  This  roadmap  was  initiated  at  the  LREC 
2002  Workshop  on  QA  and  we  propose  to  update  the  roadmap  during  the  workshop  with 
moderated  group  brainstorming  sessions. 

Characterizing  Question  Answering  Systems 

Figure  1  characterizes  a  range  of  characteristics  of  question  answering  systems.  The  set 
of  dimensions  the  distinguish  various  question  answering  systems  which  might  range 
from  systems  for  on-line  help  to  access  encyclopedic  or  technical  manual  information,  to 
open  web-based  question  answering,  to  very  sophisticated  QA  in  support  of  business  or 
military  intelligence  analyses.  Characteristics  that  distinguish  QA  environments  include 
but  are  not  limited  to: 

the  nature  of  the  query,  including  the  question  form  (e.g.,  keyword(s),  phrase(s),  full 
question(s))  the  question  type  (e.g.,  who,  what,  when,  where,  how,  why,  what-if),  and 
the  intention  of  the  question  (e.g.,  request,  command,  inform), 
the  level  of  complexity  of  the  question  and  answer, 

characteristics  of  the  source(s)  and/or  supporting  corpora  (e.g.,  size,  dynamicity, 
quality), 

-  properties  of  the  domain  and/or  task  (e.g.,  degree  of  structure,  complexity), 
the  potential  for  answer  reuse, 

the  degree  of  performance  required  (e.g.,  precision  and  recall), 

the  nature  of  the  users  (e.g.,  age,  expertise,  language  proficiency,  degree  of 

motivation)  and  the  important  of  usability, 

the  purposes  of  the  users  (e.g.,  help  with  homework  or  cooking,  strategic  analysis). 


*  This  effort  was  in  part  supported  by  the  Northeast  Regional  Reseaeh  Center  (NRRC)  whieh  is  sponsored 
by  the  Advaneed  Researeh  and  Development  Aetivity  in  Information  Teehnology  (ARDA),  a  U.S. 
Government  entity  whieh  sponsors  and  promotes  researeh  of  import  to  the  Intelligenee  Community  whieh 
ineludes  but  is  not  limited  to  the  CIA,  DIA,  NSA,  NIMA,  and  NRO. 
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nature  of  supporting  knowledge  sourees  (e.g.,  degree  of  neeessary  linguistie,  world 
knowledge) 

reasoning  requirements  (e.g.,  inferenee  required  for  question  analysis,  answer 
retrieval,  presentation  generation) 

the  degree  of  multilinguality  and  eross  linguality  (e.g.,  questions  might  be  asked  in 
one  language  ), 

the  user  model  (e.g.,  stereotypieal  vs.  individualized  user  models) 
the  task  model  (e.g.,  struetured  vs.  unstructured  tasks) 

the  type  of  answers  provided  (e.g.,  named  entities,  phrases,  factoid,  link  to  document 
summary) 

the  nature  of  interaction  (e.g.,  user  reactivity,  mixed  initiative,  question  and  answer 
refinement,  answer  justification) 


Figure  1  distinguishes  question  answering  systems  by  various  characteristics.  For 
example,  we  can  have  QA  from  a  selected  document  collection  as  in  the  Text  Retrieval 
(TREC)  QA  track,  retrieval  of  answers  from  semi-structured  sources  such  as  dictionaries, 
encyclopaedia  or  fact  books,  QA  from  massive,  unstructured  sources  such  as  the  web, 
and  multimedia  QA.  As  Figure  1  shows,  there  is  a  range  of  question/answer  complexity, 
corpus  volume,  and  degree  of  answer  integration.  Systems  may  address  a  variety  of 
question  forms  (e.g.,  keyword,  phrase,  question)  and  types  (e.g.,  who,  what,  why). 
Questions  might  encode  a  range  of  intentions  such  as  a  request  for  information,  a 
command  to  perform  some  action  such  as  a  calculation,  or  also  even  information  within 
the  question  (e.g.,  “What  type  of  Titleist  balls  does  Tiger  Woods  use?”).  The  answers 
might  come  in  the  form  of  a  named  entity,  a  phrase,  a  factoid,  a  link  to  a  document  or 
documents,  or  a  generated  summary.  Additional  characteristics  include  the  degree  of 
world  knowledge  in  the  system,  its  use  of  context  and  support  for  QA  dialogue,  if  it  has  a 
user  model  and  its  nature  (e.g.,  stereotypical,  individualized,  overlay),  its  task  model,  the 
structure  of  the  domain,  the  degree  of  answer  reuse  in  the  system,  and  the  degree  of 
expected  performance. 
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Figure  1.  Question  Answering  Characteristics 

Question  Answering  Roadmap 

Figure  2  is  a  roadmap  jointly  created  by  participants  of  the  LREC  Q&A  workshop.  The 
roadmap  is  divided  into  three  lanes  dealing  with  resources  necessary  to  develop  or 
evaluate  QA  systems,  methods  and  algorithms,  and  systems  (including  their  performance 
and  evaluation).  The  roadmap  starts  now  and  runs  until  2006.  Each  lane  leads  to 
outcomes  (indicated  by  sign  posts)  such  as  measurable  progress  from  having  shared 
resources,  a  composable  QA  toolkit,  and  personalized  QA.  An  overall,  long  term 
outcome  of  QA  systems  that  become  high  quality  and  enhance  productivity. 

Sign  posts  along  the  road  indicate  intermediate  outcomes,  such  as  a  typology  of  users,  a 
topology  of  answers,  a  model  of  QA  tasks  (from  both  a  system  and  user  perspective),  QA 
reuse  across  sessions,  and  interactive  dialogue.  Roadblocks  along  the  way  include  the 
need  to  manage  and  possibly  retrain  user  expectations,  the  need  for  reusable  test 
collections  and  the  need  for  evaluation  methods.  Overall  workshop  participants  felt  that 
general  natural  language  processing  and  inference  were  limiters  to  progress,  and  so  these 
were  represented  as  speed  limits  signs  on  the  left  hand  side  of  the  road  map.  Flere  also  we 
can  see  an  arrow  that  indicates  that  feasibility  testing  and  requirements  determination  are 
continuous  processes  along  the  road  to  productive,  quality  QA. 

On  the  right  hand  side  of  the  road  map  we  can  see  the  progression  of  question  and  answer 
types.  Questions  progress  from  simple  factoid  questions  to  how  to  why  then  to  what-if 
questions,  whereas  answers  start  out  as  simple  facts  but  move  to  scripted  or  templated 
answers  and  then  progress  further  to  include  multimodal  answers. 
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Related  fields  sueh  as  high  performance  knowledge  bases  (HPKB),  topic  detection  and 
tracking  (TDT),  databases,  virtual  reference  desks,  and  user  modeling  were  noted  as 
having  particular  importance  for  solving  the  general  QA  problem  which  will  require  cross 
community  fertilization.  Individual  activities  within  the  lanes  are  either  currently 
planned  or  future  desired  events  progressing  toward  longer  term  objectives. 
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Figure  3.  Question  Answering  Roadmap 


Future 

A  workshop  on  multilingual  summarization  and  question  answering  was  planned  at 
COLING  in  Taipei  in  August  and  a  Japanese  NTCIR  Q&A  workshop  is  being  planned 
together  with  a  future  release  of  a  Japanese  QA  corpora  (see 

research.nii.ac.jp/ntcir/workshop/qac/cfp-en.html).  We  intend  to  publish  this  roadmap 
and  regularly  update  it  as  new  resources  and  tools  emerge  and  as  new  QA  challenges 
emerge. 
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